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PREFACE 



This publication offers suggestions that may be useful in the development of plans 
for evaluating educational programs for Spanish-speaking pupils. It does not present 
a coniprehensive description of research methods nor does it encompass all 
techniques and all approaches recommended by experienced investigators. However, 
it does draw attention to the many variables that exist within any educational plan, 

and it does provide a simplified framework for exaihining such vaiiables: 

Because of the demand for copies of this publication that exceed the first printing, 
we are pleased to reissue it in this second edition. 

THOMAS A. SHELLHAMMER , .EUGENE GONZALES 

Deputy Superintendent for . Associate Superintendent 



Programs 



AofPublicInstmct^^^ 

Chief, Division of Instruction 





Some Suggestione for Evaluating the Effectivenesn of Programs 
Designed to Improve the Edueation of Mexican Americans 

The educational needs of children from Spanish-speaking families have been discussed and 
debated by educators and other interested persons for several years. Recently, renewed concern and 
available funds have resulted in the establishment of various programs which have as a primary 
objective the improvement of educational opportunities for Spanish-speaking pupils. From this 
statement of common purpose, which appears to be a unifying theme of the many efforts, the 
programs have taken diverse directions, depending upon each program planner’s definition of 
improvement. For some schools, improved opportunity has been equated with increased emphasis 
on the learning of English; for other schools, improved opportunity has been sought through an 
additional exposure to the dominant Anglo culture; for still others, improved opportunity has been 
tied to a schedule of activities that should enhance self-worth; for a few, improved opportunity has 
been seen as a concentrated program of remedial reading. Some plans have elements of all these 
curricular adjustments. There are probably as many definitions of “improvement of educational 
opportunities for Spanish-speaking children” as there are people who have anxieties about the 
progression of these pupils through a school system, which has been described at times as alien to 
them and to their families. 

In the face of so much diversity among philosophies, it is difficult for designers of programs to 
reach agreement. However, perhaps it is not necessary that there be agreement among them, for as 
ideas are exchanged, positions are defended, and viewpoints are shared, better school practices for 
teaching Spanish-speaking pupils may be developed. However, one requirement appears to be an 
essentia! ingredient of all plans and the one referential point upon which all planners can agree: the 
need for a careful appraisal of the effectiveness of programs designed to improve the educational 
opportunities of these children and young adults. Such programs should have a precise statement of 
the program’s educational purposes; a detailed description of the plan of action, including 
personnel, materials, procedures, time allotments, and other significant aspects of the proposed 
treatment; a complete listing of the data gathering techniques and instruments; a suggested outline 
of the methods of data analysis; and a careful reminder of any limitations that can be expected in 
an investigation of so many complex variables. 

On the basis of such evidence as has been gathefed arid those inferences as have been made, some 
conclusions may be drawn regarding the extent to which the stated objectives of any program have 
been achieved. For the children, the question is not merely one of improved educational 
opportunity, rather it is onC; of improved educational performance. Continuing, appraisal is needed 
to identify those. opportunities winch do result in better pupil perforrriance and those .whichTail to 
bring about the desired improvement. Empirical investigations may help estimate fhe,; worth of 
present efforts and may suggest the direction of future plans. Some recommendations for the . study 
of program efrectiveness are offered in this publication. 




Statement of the Problem 

A statement of the problem should include the reasons for the proposed development of the 
special prqgra[rii. Essentially, the problem definition not only provides a’ Rationale but also gives a 
description of the needs as identified. The meeting of these specified needs'then becomes a basis for 
the program and its operational objectives. Orie^^ such pfbblerh-as related to the concern for some 
groups of Mexican-American pupils may be. stated aa follows: , . 

Ghildren whose native language is Spanish and whose cultural backgrounds, are different from the traditional 
school culture upon which the curriculum is based may be expected to experience difficulty in meeting the demands 
of the schi^l wqrld,^i^pngT^! possib)rcauseJMfa 
may grow; (2) an inadequate command of the; English I'anguage, which 

n) a lowered self-confidence resulting frorri repeated frustration' arid failureT (4) an iurifealisti’c cufriculiirh which 
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imposes reading and writing requirements in English before skills in listening comprehension and in speaking fluency 
have been accomplished. 

In view of these factors, which are generally descriptive of the school problems of the Spanish-speaking child 
from a low-income family, it seems reasonable to design a program of instruction that would attempt to deal with 
these failure-producing conditions and to assess the effectiveness of such efforts to improve the educational 
opportunities of the Spanish-speaking children in the schools. 

Other statements to describe or to delimit any given problein may include conditions that arise 
from local needs and specific concerns as existing within a particular school. The problem elements 
as specified then may lead to the next step which requires the sifting of any applicable information 
currently available as a result of other investigations of similar problems, A review of the past 
history provides valuable background for thinking about alternative solutions to the present 
problem. 



Related Research 

The background of information generally consists of a brief review of the related research that 
has already taken place and may express the findings or conclusions of others who have studied the 
same area. As the relevant literature is reviewed and as the data given or the theories advanced are 
examined, a reasonable theoretical framework can be constructed to support the proposed 
manaiement of the educational problem. This framework then forms the basis upon which the 
hypotheses of the investigation can be made. 



Literature Review 

To cover the background information needed for an exploration, of the several aspects of the 
problem previously cited, a review of the literature would probably include a survey of definitions, 
findings, and conclusions pertaining to the following: (1) concept formation and acquisition; 
(2) theories of bilingualism and its effect upon cognitive development; (3) theories of self-concept 
and the influences of frustration or failure upon self-worth; (4) the second-language learning 
theories, particularly learning English by native speakere of Spanish. Each, of these four major 
variables could probably be explored in detail through an examination of in ppfessional 

journals and books in which any previous applicable research has beeh reported. 

A review of the literature is generally a time-consuniing task and one that demands great patience 
and care. The extracted information, however, may provide additional insights and greater 
understanding of the sfated pS^blem as weU^^^^ solutions which 

may be tested, Forithese reasons, this’ searcl^ng of any available w'ritings on the subject is likely to 
be one of the most valuable aspects of any study. 



Presentation of Backpound Information 

A few examples may illustrate a way of presenting the background information: 

Concepts. Jerpme Brunet, and others in their volunie, A point out . that the. basis for all 

thinking and cognition is concept ‘formation and attainment,* Gage Has suggested that there are at least three 
different ways in which new concepts may be formed and that these three ways may not be mutually exclusive. He 
describes, these ways as; (1) the perceiving of common characteristics; (2) the corabirung of concepts already known, 
and (3) the inferring of underlying states or constructs, as a means of ' forming concepts that are hot directly 
observable.^ . • , , 



•jerome S. Bruner, JaequeUne Goodnow, and G. A. Austin-d o/rAinWnf.;New,York: John WUey & Sons, Inc., 1956. 
^N. L. Gage. //flnrfhooko/ResflffrcA on T’cccft/ng, Chicago: Rand McNally &,Co„ 1963. 
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Bilingualism. Children who live in two cultures with languages that are different can be expected to develop some 
degree of bilingualism. Weinreich has said that bUingudism may be defined in several ways.^ One theory of 
bilingualism is advanced by Emn and Osgood, who suggest that the kind of bilingual system acquired depends upon 
the situation in which the language has been learned. If two languages are learned in a single situation or context and 
the native language is used to mediate meaning, then a compound system results. If the two languages are learned in 
two different situations, a coordinate system develops,^ 

Seif concept. Loevinger reports that there is a gradual increase in the tendency to see one’s self faborably between 
the ages of eight and thirteen.^ Sears and Sherman (1964) studied the development of self-esteem in school children 
and defined self-esteen as a favorable concept of self. TTiey identified areas of experiences that appear to be the 
sources for children’s perceptions of selfesteem.^ 

English as a second language. Ott has said that there are two main problems in second language learning. The first 
one exists in the relative difficulty of initial learning which may be decreased as the student becomes familiar with 
the language, and the second problem may be found in the learning of many vocabulary items and grammar rules 
without accomplishing speaking fluency J 



Documentation of References 

It is essential that summarized information be carefully documented for author, title, publisher, 
and publication date so that anyone who wishes to consult the original writing may find the source 
from which the summaries have been made. 

The Investigation 

The study should identify (1) the pupils who are to be the subjects of the investigation; (2) the 
educational plan that is to be applied; (3) the period of time that the experiment will take; (4) the 
comparison groups that may be participating in the study; (5) the methods and instruments by 
which XI - data will be gathered; (6) the means by which the data will be analyzed; (7) the 
procedures by which the significant variables will be controlled; and (8) the predictions or outcomes 
expected as stated in the hypothesis. 

Subjects to Be Studied 

The pupils whose school problem is under investigation should be described in careful detail. The 
desenption should include the total humbar of pupik and girls, their 

chronological ages, the grade placements, their language backgrounds in both Spanish arid Englis^^ 
their socioecohomic status, and any other relevant data. Such matters as rural or urban living, 
occupation of the father, educational level of the mother, place in the birth order^ 
transfer, and such information as seems necessary to provide a clear* picture of the attributeV^^^^^^ 
pupils should be provided. For example, one description of the subjects might be the following: , 

Nineteen pupils in the Mary CovUlaud School, Marysville, California, will be the subjects of this investigation. 

They are children for whom Spanish is a first langua|e and for whom En^ish is a second one. The group is cdrhposed 

. ■ V ■ ■ - ■ = 

^Uriei WeiniBich. Confaef. New York: Humanities Press, Inc*, 1953. , . j ^ v ; : ! 

/^Susan Ervin and C. E. Osgood. Qted in Gage, op. dL, p. 1,085.; ; L : • ^ ^ :: 

^Jane Loevinger. Cited in Paul H. Mussen. Handbook of Research Methods in Child Development. New York: John WUey & Sons, 

Inc.. 1960. : 

^Pauline S. Sears and Vivian S. Sherman. In Pursuit of Self-Esteem. Belmont, ..Calif.: Wadsworth riibUsIwg Comply, Inc., 1964. 

■^Elizabeth H. Ott. A Study of Levels of Fluency and ^oficiency in Oral English of Spanish-Speaking School Beginners. Austin. , 

Texas: The University of Texas, 1967. '■ ^ 
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of nine boys and ten girls whose grade placement ranges from kindergarten through grade three. There is a four-year 
age span ftom 6.0 to 10.1 years. All pupils have very limited or no command of English. They come from 
low-income homes in which Spanish is spoken by their parents. These boys and girls have had few opportunities for 
contact with the dominant culture of the community. 



Procedure to Be Followed 

The procedure may outline the proposed educational plan, its purposes, and how it is to be 
applied. The objectives should be specified in operational terms and should be derived from the 
statement of the problem and from the solutions suggested as a result of the review of the literature. 
In the case of a problem stated as a language-learning deficit in English, and with the thesis 
advanced that oral proficiency should precede reading and writing skill development, one aim of the 
program could be to increase the number of words and utterances in English that the pupils could 
understand and say. Other goals might also be specified as the use of complete sentences, the 
knowledge of correct word order, an awareness of varying patterns of intonation and stress, and a 
comprehension of certain structural variations in English. 



A precise statement of expected outcomes provides direction for the important components of 
the educational plan: personnel, materials, techniques, methodologies, time schedules, supportive 
services, school plant, and other curricular matters. Those features of the plan which represent a 
departure from the usual schooE management of these pupils should be pointed out and fully 
explained. The addition of different courses, new people and their functions, health services, 
increased time allotments for special subject matter, inservice training of teachers, parent education, 
special organization of classes, and any other facet of school or community support that is being 
provided for the improvement of educational opportunities should be described. 



Special effort must be made to differentiate from among the rnany strands of the curriculum 
those provisions which would have been made as paurt of an ordihary program and those 
arrangements which represent changes specifically applied for the purposes stated. Eor instance, if it 
has been hypothesized that a program of bilingual iristructibrj using both Spanish and English w 
result in improved achievement in arithmetic and increased fluency, 

description of the plan should state what is being done about teaching the pupils what 

approach is being taken in the teaching of En^ish, and by what criteria achieveinent in arithmetic 
and fluency in English are to be evaluated. An illustration follows: 


















. . .. ... ? ;■ 

'• V " '.V'’ .1..- • 

The chUdren wUl spenjd ey^ morning in a special M 

good command of Spanish: She is assisted a teacher aide, a native^spe^er of adequate command 

of English. The room will be attractively furnished ^d.equipped with a variety of bodies, games, toys, aft materials, - 
manipulative devices, and records. Pictures, charts, and,jopm_ decorations that contam^^writmg will pe 
Spanish. Concepts in arithmetic wfil be developedrbn a edherete level supported by use 
rhediate meaning until such time as' the pupils have buUt an arithmetic vocabulary in A' daily pfdg^^ 

develop and to extend the "pupU^ ^ 

program of English-as-a-second-language Will -be giveh. The scope, ^ sequence, and pace of presentation will - be* 
carefully controlled and sysfematicaily reviewed to ensure sufficient repetition and practice which will lead to ' . 

For part of each day . the pupils wUi participate in othat classroom activiti of the Englishfspeaking children. In : 
music, art, and physical .education classes where language is not a hindrance to.meaning, the Spanish-speaking pupils 









'ffi- 






wdl join the various groupl according to their most appropriate' age placement They' will" also share in the^'total life 

of the school by having the sanieTecess times for informal playground ‘activities' and by taldiig'paft 'ih -iassemblies, 

1,,; ■ - : . ^ .... ’ - ' 






scKoblwide parties,.and 


















iff 



Growth in iproficiency in ‘English' and -achievementt in arithmetic/ wUi; be -measured: ‘A^description of the 



measurement'iristrurhents 



wai:beprovided'inThe‘¥e'cTi6h’Ondata%ath'ering techniques;^ 
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Time Period to Be Used 



The plan may be applied for any period of time desired. The usual length of time is the regular 
school year from September to June. If the number of pupils is large enough and if a high rate of 
pupil transfer does not occur, the time could be extended over several school years, possibly a two- 
or three-year period. Such a longitudinal study would require very careful recognition of all the 
variables stated as well as others that might result from the passage of time and changes in 
developmental states of the pupils. The infomation yielded could be rich in ideas for planning both 
short- and long-range goals. The time period may also include the schedule of testing, interviewing, 
or other assessment techniques so that the effects of the educational plan, in terms of adjustment 
and enrichment, may be assessed. The most common arrangement provides for an appraisal before 
the program is offered and one after it has been provided. These before and after assessments are 
usually done in the fall and in the spring of the school year. 



Comparison Groups to Be Established 

It is often suggested that poups for whom no special program is being provided be included in 
the study so that the results can be interpreted against a background of comparable variables. 
Subjects in comparison poups should be described in detail, and the criteria for selection should be 
clearly stated. Pupils chosen for comparison with the subjects mentiohed earlier may be identified 
in the following manner: 



Nineteen pupils in another school in the district represent the same chronological age, sex, socioeconomic status, 
ethnic background, and language characteristics as those pupils identified as subjects of the investigation. No 
systematized attempt to provide special instruction in English or in arithmetic is being made, niese pupils are in a 
regular program in which all the instruction is given in English, No Spanish is used in the classroom nor is the 
program in English organized as a second-language approach, Reading, writing, w taught in English for 

all children, with a grouping proyision for “slower” pupils within the class. ; ' 

The comparison group may be tested and intemewed at the same intervals and with the same 
instruments as used for the group in the special program. The schpoi progress of both groups may 
then be compared and the differences analyzed. , ; ^ . : 1 * . 




Methods of Study to Be Employed 



Data about both groups of chUdreh — ; those in the special program^arid 'those being used for 
comparison purposes — may be gathered by interviewing parents and « using a variety of measuring 
instruments. ' . . 



- Intehiews witH pdreriti. Home visits jhay 
health nurse, who is a nfiember of the health 






school employee. To provide cbnipafabinty,J6f. thfe':ihf 0 rniariomgm^^^^ 



intervie w tebhriique may be used Each farhily' may be asked! to respond to questions' concerning the 



following: 



• Parents: (I) educational level of the p^ countryiof their birth; ^ 

af home; (4) bccupatidh and, employment st^^^ home owhership^OT 

; the.pareht^^ and (7),any pther.gen'efM in^^ ' f- ■ r * > . r : * . 













Other information obtained during the interview may be included if it is pertinent to the study. 



Measuring instruments. Assessment of intellectual polientlal vocabulary level in Spanish and in 
English may be accomplished by the use of the following instruments: 



1. Escala de Inteligencia Wechsler para ninos is a translation and adaptation of the “Wechsler 
Intelligence Scale for Children” (WISC). * he WISC consists of 12 subtests that attempt to 
determine areas of strength and weakness. Scores are reported for the verbal and nonverbal 
abilities as well as in full-scale score reported in both scaled scores and I.Q. 

2. Directions may be given in Spanish for the Peabody Picture Vocabulary, The age range of the 
Peabody is from three through eighteen yeare. The pupil is required only to point at the 
response of his choice. Items are arranged in order of difficulty to penriit basal and a ceiling 
age. Buros reports moderate reliability and advises caution in using the norms. Correlations 
with WISC and Binet appear to be in the 70s and low 80s. 



3. Three Wishes is a projective technique to attempt to draw from the pupils some information 
regarding their needs, motives, dreams, and desires. "St tuvieras tres deseos dcuales, serian?" 

4. The Madeline Thomas Completion Srones is projective technique for children which uses a 
series of stories without endings. The stories may be adapted in Spanish and read to the pupil. 
An oral response from the child may be elicited. The content is that which may be of common 
concern to school children of any cultural group or social class. There are two levels of The 
Madeline Thomas Completion Stories for both elementary and high school pupils. 

5. The children are asked to draw a person in the Machover Draw-a-Person. Directions may be 
given in Spanish, and a few questions about the drawing may be asked : 

6. The Bender-Gestalt Test for Young Children is a perceptual and projective test for children 
between the ages of five and ten years. It consists of nine figures, which are presented one at a 
time. The pupils are asked to copy these designs on a blank piece of paper. Scoring and 
interpretation of the drawings Will yield information on the visual-motbr perceptual skills of 
the children and some clues to their pemonality traits. The regular 5ender-Gesra/r Test may be 
used with older pupils by using the regular scoring and administration techniques. 



7. The GeseU Maturation Index is an index of school readiness. Ghdbreii are; given various 
developmental tasks and are assi^ed a; maturation levef on the ^basis : of : their : success ;in 
managing the various tasks. This instrument is very useful in the, appraisarof ypung children. 

8: The Lin^istic Capacity ^ Index is a test based on a contrastive analysis of grammar and 
:: phonology Kin; both v: English: and Spmish.flt is :a: ^oupameasure administered .ofallj^^hy . the 
examiherrThe' pup^^^ responds by plachig a mark ph a picture. The suhfert 
. ; contrastive' phopolq^^ and contrastive grammar. It was .designed ,.tQ Test the Englishdanguage 
: skills of pupils whose native language is Spanish and assumes some underarand/ng of English. It 
does riot test dm/ English. ' . 



Analysis of Data 






:->r: :-‘r ■- c :' \ 

The data gathered about bo^^^ of children"^ thosejn the special «p^^ 

comparison group — may serve a descriptive puipose to identify the childrens* learning potentials, 



their.:strengths.iand.,. 
disebvefigrov^hipattOT 





special pro^am . » Data- for boys and«girls;.shquld be 



edmpare^^ 



■■ :>r. -•? 






„ 
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analyzed separately. Appropriate statistical measures may be applied to test the significance of the 
differences if any are indicated. Case studies may be written in detail for each child if the number of 
pupils is not too large and if the evaluator has enou^ time and assistance to complete the individual 
case studies. 



Limitations of the Investiption 

There may be several limitations in any investigation that suggest caution in applying the findings 
to other groups. First, if the number of subjects is small, a wide application of the findings cannot 
be assumed. In addition the teacher variable should be noted. If there is one teacher and one teacher 
aide involved in the instmctional program that is being examined and if there are several teachers 
and several aides in the comparison group, the variables among teachers in training experiences, 
attitudes, and other important aspects of classroom management may not have been controlled. 
One must also consider the fact that the “Hawthorne Effect” may have influenced the performance 
of the special educational plan. If the investigation proposes to look carefully at pupils in the early 
primary grades, the generalizations drawn from such a study would not necessarily apply to children 
in other grades. Finally, there are some cultural biases in the measuring instruments that are not 
removed by simply translating English words into Spanish. The WISC, for example, has been 
standardized on a Puerto Rican population, and most of these children are of Mexican background. 
Vocabulary tests, regardless of language used to administer them, are likely to favor middle-class 
children whose experiences are generally wider than those of children who come from low-income 
families. 



In spite of these many stated limitations, the data gathered and analyzed should provide valuable 
descriptive information and suggest several implications for thb improvement of the educational 
opportunities for Spauiish-speaking chUdren. The data^^^:^^ 

question: To what extent did the program realiy provide improved opportunities for 
Spanish-speaking pupils, and to what extent has the educational performance of these pupils been 
improved? 



This Publication provides an overview of the miiy elements which should be included in an 
evaluation of a program designed to improve education^i'opportunities of Spanish-sbeakih^ 
of Mexican-American heritage. It does not represeht a rigdrbus ridf a hi^ly sophisticated approach 
to research design. The suggestions are offered merely as a way of appraising the m 
are presently: to of Jhese.;pupils. N. L. G&bs^s Haridbook of Research on 

Teaching and Paul H. Mussen’s Handbook of Resemch«Methods in%ChUd Development aU&r 
research models and excellent advice on measurement and evaluation. 



Jose Enrique Rodo, a great intellectual and spiritual voice of Latin America at the turn of the 
century, said, “A man who is carried away by. the partial appearance of things, and whose point of 
view is in error, ir heading straight for mediocrity;"^ ; ^ 



The search for a more complete picture of the educational needs of Spanish-speaking children 
and for improved methods of meeting such needs must be continued. For, as Rodo has warned, the 



dangers implicit in a “partial appearance of things” are self-evident. 
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